[opt](scanner) Control the degree of parallelism of scanner when only limit involved #39927
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thank you for your contribution to Apache Doris. Since 2024-03-18, the Document has been moved to doris-website. |
Contributor
|
clang-tidy review says "All clean, LGTM! 👍" |
yiguolei
reviewed
Aug 26, 2024
yiguolei
reviewed
Aug 26, 2024
yiguolei
reviewed
Aug 26, 2024
yiguolei
reviewed
Aug 27, 2024
| description = { | ||
| "请使用 adaptive_pipeline_task_serial_read_on_limit", | ||
| "Please use adaptive_pipeline_task_serial_read_on_limit"}) | ||
| public long limitRowsForSingleInstance = 8192; |
Contributor
There was a problem hiding this comment.
保持原来的值，10000就行了，万一有一些极限情况，上线后，直接性能下降了
Contributor
Author
There was a problem hiding this comment.
FE BE上参数不一样，都改成10000？
zhiqiang-hhhh
force-pushed
the
opt-session-scanner-limit
branch
from
6c4ce1b to
8381b41
Compare
Contributor
Author
|
run buildall |
Contributor
|
clang-tidy review says "All clean, LGTM! 👍" |
zhiqiang-hhhh
changed the title
Contributor
|
clang-tidy review says "All clean, LGTM! 👍" |
TPC-H: Total hot run time: 38150 ms |
TPC-DS: Total hot run time: 193246 ms |
ClickBench: Total hot run time: 31.16 s |
Contributor
Author
|
run buildall |
Contributor
|
clang-tidy review says "All clean, LGTM! 👍" |
TPC-H: Total hot run time: 38335 ms |
TPC-DS: Total hot run time: 193613 ms |
ClickBench: Total hot run time: 31.93 s |
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 38507 ms |
TPC-DS: Total hot run time: 193612 ms |
ClickBench: Total hot run time: 32.91 s |
ClickBench: Total hot run time: 32.77 s |
yiguolei
approved these changes
Sep 4, 2024
zhiqiang-hhhh
added a commit
to zhiqiang-hhhh/doris
that referenced
this pull request
Sep 4, 2024
… limit involved (apache#39927) For queries like `select * from tbl limit 100;`, we should limit the parallelism to 1. We already have strategies in our code, but we can not control its behavior. So add session variables to control them. Previous pr like apache#33888 apache#36535 already has done something similar, this pr integrates them.
This was referenced Oct 8, 2024
yiguolei
pushed a commit
that referenced
this pull request
Oct 9, 2024
enable_scan_node_run_serial is not working after #39927
eldenmoon
pushed a commit
to eldenmoon/incubator-doris
that referenced
this pull request
Oct 10, 2024
enable_scan_node_run_serial is not working after apache#39927
cjj2010
pushed a commit
to cjj2010/doris
that referenced
this pull request
Oct 12, 2024
enable_scan_node_run_serial is not working after apache#39927
amorynan
pushed a commit
to amorynan/doris
that referenced
this pull request
Oct 12, 2024
enable_scan_node_run_serial is not working after apache#39927
morningman
pushed a commit
to apache/doris-website
that referenced
this pull request
Oct 16, 2024
# Versions - [x] dev - [x] 3.0 - [x] 2.1 - [ ] 2.0 # Languages - [x] Chinese - [x] English apache/doris#39927
zhiqiang-hhhh
added a commit
to zhiqiang-hhhh/doris
that referenced
this pull request
Oct 17, 2024
enable_scan_node_run_serial is not working after apache#39927
zhiqiang-hhhh
added a commit
to zhiqiang-hhhh/doris
that referenced
this pull request
Oct 18, 2024
… limit involved (apache#39927) For queries like `select * from tbl limit 100;`, we should limit the parallelism to 1. We already have strategies in our code, but we can not control its behavior. So add session variables to control them. Previous pr like apache#33888 apache#36535 already has done something similar, this pr integrates them.
yiguolei
pushed a commit
to zhiqiang-hhhh/doris
that referenced
this pull request
Oct 19, 2024
… limit involved (apache#39927) For queries like `select * from tbl limit 100;`, we should limit the parallelism to 1. We already have strategies in our code, but we can not control its behavior. So add session variables to control them. Previous pr like apache#33888 apache#36535 already has done something similar, this pr integrates them.
zhiqiang-hhhh
added a commit
to zhiqiang-hhhh/doris
that referenced
this pull request
Nov 8, 2024
… limit involved (apache#39927) For queries like `select * from tbl limit 100;`, we should limit the parallelism to 1. We already have strategies in our code, but we can not control its behavior. So add session variables to control them. Previous pr like apache#33888 apache#36535 already has done something similar, this pr integrates them.
zhiqiang-hhhh
added a commit
to zhiqiang-hhhh/doris
that referenced
this pull request
Nov 11, 2024
enable_scan_node_run_serial is not working after apache#39927
16 tasks
zhiqiang-hhhh
added a commit
to zhiqiang-hhhh/doris
that referenced
this pull request
Dec 27, 2024
Related PR: apache#39927 apache#39927 introduced a new session var `adaptive_pipeline_task_serial_read_on_limit`, its implementation has flaw, reduce scanner parallelism should only happen when there is not conjunction. Since we can not estimate the filtration efficiency.
zhiqiang-hhhh
added a commit
to zhiqiang-hhhh/doris
that referenced
this pull request
Dec 27, 2024
Related PR: apache#39927 apache#39927 introduced a new session var `adaptive_pipeline_task_serial_read_on_limit`, its implementation has flaw, reduce scanner parallelism should only happen when there is not conjunction. Since we can not estimate the filtration efficiency.
zhiqiang-hhhh
added a commit
to zhiqiang-hhhh/doris
that referenced
this pull request
Jan 1, 2025
enable_scan_node_run_serial is not working after apache#39927
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
For queries like
select * from tbl limit 100;, we should limit the parallelism to 1.We already have strategies in our code, but we can not control its behavior. So add session variables to control them.
Previous pr like #33888 #36535 already has done something similar, this pr integrates them.